High Test Maintenance Cost

Test code needs to be maintained along with the production code it verifies. As we evolve an application we will likely have to revisit our tests on a regular basis as the classes they verify are changed to add new functionality or be simplified through refactoring. High Test Maintenance Cost is what occurs when the tests become hard to understand and maintain.

Symptoms

Development of new functionality is slowing down. Every time we add some new functionality, we need to make extensive changes to the existing tests. Developers or testers may be telling the project manager or coach that they need a "test refactoring/cleanup iteration".

If we have been tracking the amount of time we spend writing the new tests and modifying existing tests separately from the time we spend implementing the code to make the tests pass, we notice that most of the time is spent modifying the existing tests.

Most test maintainability issues are accompanied by other smells such as:

• Fragile Tests (page X) are the symptom that indicate that tests are too closely coupled to the system under test (SUT).
• A Fragile Fixture (see Fragile Test) is a sign that to many tests depend on the same fixture design (Standard Fixture (page X) and that is leading to High Test Maintenance Cost.
• Erratic Test (page X) may be a sign that a Shared Fixture (page X) is causing us a lot of grief.

Impact

Team productivity drops significantly because the tests take so much effort to maintain. Developers may be agitating to "cut and run" (remove the affected tests from the test suites.) While writing the production code is mandatory, maintaining the tests is completely optional (at least to the uninformed). If nothing is done about this problem, the whole test automation effort may be wasted when the team or management decides that test automation just "doesn't work" and they decide to abandon the tests.

Causes

The root cause is failing to pay attention to the various principles described in Principles of Test Automation. A more immediate cause is often too much Test Code Duplication (page X) and tests that are too closely coupled to the API of the SUT.

High Test Maintenance Cost can be caused by many things.

Cause: Fragile Test

Tests that fail because minor changes were made to the SUT are called Fragile Tests. They result in High Test Maintenance Cost because they need to be revisited and "giggled" after all manner of minor changes that really shouldn't affect them.

The root cause of the failure can be any of the "four sensitivities" (Interface Sensitivity (see Fragile Test), Behavior Sensitivity (see Fragile Test), Data Sensitivity (see Fragile Test) and Context Sensitivity (see Fragile Test).) We can reduce the High Test Maintenance Cost by making the tests insensitive to as many of these sensitivities as possible through the use of Test Doubles (page X) and by refactoring the system into smaller components and classes that can be tested individually.

Cause: Obscure Test

Obscure Tests (page X) are a major contributor to High Test Maintenance Costs because they take longer to understand each time they are visited. When they do need to be modified, they take more effort to adjust and are much less likely to "work the first time" resulting in more debugging of tests. They are also more likely to end up not catching conditions they were intended to and that can lead to Buggy Tests (page X).

Obscure Tests are best addressed through refactoring of tests focusing on the reader of the tests. The real goal is Tests as Documentation (see Goals of Test Automation on page X) any anything less than that will increase the likelihood of High Test Maintenance Cost.

Cause: Hard to Test Code

"Legacy software" (any software that doesn't have a complete suite of automated tests) can be hard to test since we are typically writing the tests "last" (after the software already exists.) If the design of the software is not conducive to automated testing, we may be forced to use Indirect Testing (see Obscure Test) via awkward interfaces that involve a lot of accidental complexity; that may result in Fragile Tests or a Fragile Fixture.

It will take time and effort to refactor the code to improve its testability but that time will be well spent if it eliminates the High Test Maintenance Cost. If refactoring is not an option, we may be able to reduce the amount of test code impacted by a change by doing SUT API Encapsulation (see Test Utility Method on page X) using Test Utility Methods. For example, Creation Methods (page X) encapsulate the constructors and therefore make the tests less susceptible to changes in constructor signatures or semantics.

Trouble-Shooting Advice

As a project level smell, High Test Maintenance Cost is as likely to be detected by a project manager, Scrum master or team lead as by a developer. While we may not have the technical depth to troubleshoot and fix the problem, the fact that we are aware of it is what is important. This allows us to ask the development team questions about how long it is taking to maintain tests, how often it occurs and why it is necessary. Then we can challenge them to find a better way; one that won't result in such High Test Maintenance Costs!

They'll need our support, of course, to be able to carry out whatever improvement plan they come up with. That support will have to include the time to do the investigations (spikes), learning/training time and time to do the actual work. We can make time for this by having "test refactoring stories", adjusting the velocity to reduce the new functionality committed to the customer or some other means. Regardless of how we do it we mustn't forget that if we don't give the development team the resources to fix the problem now, the problem will only get worse and become even harder to fix in the future when we have twice as many tests.

Page generated at Wed Feb 09 16:39:52 +1100 2011

Copyright © 2003-2008 Gerard Meszaros all rights reserved